{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bLEEW13uCtiJ"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_03_5_pytorch_class_sequence.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5oX6K_-JCtiL"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 3: Introduction to PyTorch**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FzlMdsudCtiL"
   },
   "source": [
    "# Module 3 Material\n",
    "\n",
    "- Part 3.1: Deep Learning and Neural Network Introduction [[Video]](https://www.youtube.com/watch?v=d-rU5IuFqLs&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_03_1_neural_net.ipynb)\n",
    "- Part 3.2: Introduction to PyTorch [[Video]](https://www.youtube.com/watch?v=Pf-rrhMolm0&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_03_2_pytorch.ipynb)\n",
    "- Part 3.3: Encoding a Feature Vector for PyTorch Deep Learning [[Video]](https://www.youtube.com/watch?v=7SGPm2tIT58&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_03_3_feature_encode.ipynb)\n",
    "- Part 3.4: Early Stopping and Network Persistence [[Video]](https://www.youtube.com/watch?v=lS0vvIWiahU&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_03_4_early_stop.ipynb)\n",
    "- **Part 3.5: Sequences vs Classes in PyTorch** [[Video]](https://www.youtube.com/watch?v=NOu8jMZx3LY&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_03_5_pytorch_class_sequence.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CpVJpj2DCtiM"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running and maps Google Drive if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wU1AMzx8CtiM",
    "outputId": "014519db-db87-41b6-f5b6-e0a68e2fdffe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n",
      "Using device: mps\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    import google.colab\n",
    "\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "    import torch\n",
    "\n",
    "# Make use of a GPU or MPS (Apple) if one is available.  (see module 3.2)\n",
    "import torch\n",
    "device = (\n",
    "    \"mps\"\n",
    "    if getattr(torch, \"has_mps\", False)\n",
    "    else \"cuda\"\n",
    "    if torch.cuda.is_available()\n",
    "    else \"cpu\"\n",
    ")\n",
    "print(f\"Using device: {device}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1Zg-zNx0CtiN"
   },
   "source": [
    "# Part 3.5: Sequences vs Classes in PyTorch\n",
    "\n",
    "Previously, we explored the fundamentals of PyTorch and how to define neural networks using the **nn.Sequential** module. This approach allowed us to build neural networks by specifying a sequence of layers straightforwardly and concisely. While **nn.Sequential** is the primary method we will use in this course, it is essential to understand an alternative way of defining neural networks as a class.\n",
    "\n",
    "## Defining Neural Networks as Sequences\n",
    "\n",
    "Before we review the class-based approach, let's recap how we define neural networks using the **nn.Sequential** module. In PyTorch, you represent a neural network as a sequence of layers. Each layer performs a specific computation on the input data and passes the transformed output to the next layer.\n",
    "\n",
    "Here's a quick example of defining a simple neural network using **nn.Sequential**:\n",
    "\n",
    "```\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# Define the neural network architecture\n",
    "model = nn.Sequential(\n",
    "    nn.Linear(784, 128),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(128, 64),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(64, 10),\n",
    "    nn.LogSoftmax(dim=1)\n",
    ")\n",
    "\n",
    "```\n",
    "\n",
    "In this example, we created a neural network with three linear layers interspersed with ReLU activation functions. The final layer utilizes the **LogSoftmax** activation, which we commonly use for classification tasks.\n",
    "\n",
    "## Defining Neural Networks as Classes\n",
    "\n",
    "While the nn.Sequential approach is intuitive and convenient for many cases, it can become limiting when we need more flexibility in the network architecture. Defining neural networks as classes allows us to create custom architectures with complex behaviors and shareable components.\n",
    "\n",
    "To define a neural network as a class, we typically subclass the **nn.Module** class provided by PyTorch. This base class offers essential functionality for organizing the network's parameters and handling computations during forward and backward passes.\n",
    "\n",
    "Let's illustrate this by rewriting our previous example using a class-based approach:\n",
    "\n",
    "\n",
    "```\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class CustomNetwork(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(CustomNetwork, self).__init__()\n",
    "        self.fc1 = nn.Linear(784, 128)\n",
    "        self.relu1 = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(128, 64)\n",
    "        self.relu2 = nn.ReLU()\n",
    "        self.fc3 = nn.Linear(64, 10)\n",
    "        self.log_softmax = nn.LogSoftmax(dim=1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu1(x)\n",
    "        x = self.fc2(x)\n",
    "        x = self.relu2(x)\n",
    "        x = self.fc3(x)\n",
    "        x = self.log_softmax(x)\n",
    "        return x\n",
    "\n",
    "# Create an instance of the CustomNetwork\n",
    "model = CustomNetwork()\n",
    "\n",
    "```\n",
    "\n",
    "In this example, we define a class called CustomNetwork that inherits from nn.Module. Inside the class, we describe the network's layers as attributes. The forward method specifies how the input flows through these layers during the forward pass.\n",
    "\n",
    "By defining neural networks as classes, we can create more complex architectures, leverage conditional logic within the network, and encapsulate reusable components. This flexibility becomes particularly useful as we delve into advanced topics throughout the course.\n",
    "\n",
    "## Choosing the Appropriate Method\n",
    "\n",
    "While both the nn.Sequential and class-based approaches allow us to define neural networks; we will primarily utilize nn.Sequential in this course due to its simplicity and convenience. It offers an efficient way to construct networks with a linear sequence of layers.\n",
    "\n",
    "However, understanding the class-based approach is crucial for deeper customization and building more intricate architectures. As you progress in mastering PyTorch, you'll encounter scenarios where defining networks as classes becomes necessary. You'll be well-equipped to tackle a wide range of neural network design challenges by grasping both methods.\n",
    "\n",
    "In the upcoming chapters, we will explore various applications of nn.Sequential and learn advanced techniques to enhance the performance and capabilities of our neural networks.\n",
    "\n",
    "The following code presents a complete example of the MPG dataset using a **nn.Module** class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "szrd-36UE_ZA"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0, loss: 1545.567138671875\n",
      "Epoch 100, loss: 177.34703063964844\n",
      "Epoch 200, loss: 136.75787353515625\n",
      "Epoch 300, loss: 47.36888122558594\n",
      "Epoch 400, loss: 26.93931007385254\n",
      "Epoch 500, loss: 18.710189819335938\n",
      "Epoch 600, loss: 13.950730323791504\n",
      "Epoch 700, loss: 11.416263580322266\n",
      "Epoch 800, loss: 10.207803726196289\n",
      "Epoch 900, loss: 9.617264747619629\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from sklearn import preprocessing\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from torch.autograd import Variable\n",
    "\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self, input_dim, output_dim):\n",
    "        super(Net, self).__init__()\n",
    "\n",
    "        # Define each of the layers\n",
    "        self.layer1 = nn.Linear(input_dim, 50)\n",
    "        self.layer2 = nn.Linear(50, 25)\n",
    "        self.layer3 = nn.Linear(25, output_dim)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Pass the input through each of the layers\n",
    "        x = F.relu(self.layer1(x))\n",
    "        x = F.relu(self.layer2(x))\n",
    "        return self.layer3(x)\n",
    "\n",
    "\n",
    "# Load and preprocess data\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", na_values=[\"NA\", \"?\"]\n",
    ")\n",
    "\n",
    "# Replace missing horsepower values with median\n",
    "df[\"horsepower\"] = df[\"horsepower\"].fillna(df[\"horsepower\"].median())\n",
    "\n",
    "# Convert pandas DataFrame to PyTorch tensors\n",
    "x = torch.tensor(\n",
    "    df[\n",
    "        [\n",
    "            \"cylinders\",\n",
    "            \"displacement\",\n",
    "            \"horsepower\",\n",
    "            \"weight\",\n",
    "            \"acceleration\",\n",
    "            \"year\",\n",
    "            \"origin\",\n",
    "        ]\n",
    "    ].values,\n",
    "    device=device,\n",
    "    dtype=torch.float32,\n",
    ")\n",
    "y = torch.tensor(df[\"mpg\"].values, device=device,\n",
    "                 dtype=torch.float32)  # regression\n",
    "\n",
    "# Initialize the model, loss function, and optimizer\n",
    "model = Net(x.shape[1], 1).to(device)\n",
    "model = torch.compile(model,backend=\"aot_eager\").to(device)\n",
    "loss_fn = nn.MSELoss()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n",
    "\n",
    "# Training loop\n",
    "for epoch in range(1000):\n",
    "    # Zero gradients\n",
    "    optimizer.zero_grad()\n",
    "\n",
    "    # Forward pass\n",
    "    outputs = model(x).flatten()\n",
    "\n",
    "    # Compute loss\n",
    "    loss = loss_fn(outputs, y)\n",
    "\n",
    "    # Backward pass and optimize\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    if epoch % 100 == 0:\n",
    "        print(f\"Epoch {epoch}, loss: {loss.item()}\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3.9 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
